Skip to content

feat: Phase 10: Synthetic workload generation & telemetry validation suite - #6519

Open
pratikmankawde wants to merge 470 commits into
pratik/otel-phase9-metric-gap-fillfrom
pratik/otel-phase10-workload-validation
Open

pratikmankawde wants to merge 470 commits into
pratik/otel-phase9-metric-gap-fillfrom
pratik/otel-phase10-workload-validation

Conversation

@pratikmankawde

@pratikmankawde pratikmankawde commented Mar 10, 2026

Copy link
Copy Markdown
Contributor

PR Chain: Phase-1aPhase-1bPhase-1cPhase-2Phase-3Phase-4Phase-5Phase-6Phase-7Phase-8Phase-9#6519 (this PR, Phase-10)
Base: pratik/otel-phase9-metric-gap-fill
Consolidated PR: #7770

High Level Overview of Change

The validation harness for everything above: drives a multi-node validator cluster under synthetic RPC and transaction load, asserts every expected span, attribute and metric actually fires, then gates timing against a committed baseline. Test and CI assets only — no src/ or include/ changes.

No need to review tasklist files.

Context of Change

Up to here, "the telemetry works" rested on reading dashboards by hand. This adds a repeatable check.

  • Cluster + loaddocker-compose.workload.yaml, generate-validator-keys.sh, xrpld-validator.cfg.template; rpc_load_generator.py (async WebSocket, configurable rate/mix) and tx_submitter.py (10 transaction types, configurable TPS, periodic sequence refresh), sequenced by workload_orchestrator.py.
  • Assertionsvalidate_telemetry.py checks the inventories in expected_spans.json (41 span types, 62 unique attributes, plus parent-child relationships) and expected_metrics.json (26 metric groups) against live Tempo and Prometheus.
  • Regression gatecapture_timings.py queries histogram_quantile over span_duration_milliseconds_bucket and job_{queued,running}_us_bucket, producing 28 keys: 8 span names × p50/p95/p99, plus 2 job types × queued/running at p95. compare_to_baseline.py diffs them against baselines/baseline-timings.json (populated, captured 2026-06-05) and flags a regression only when both the percentage and absolute bounds are breached.
  • Excluded on purposerpc.process is not gated: it is created only on the HTTP JSON-RPC path, which the WebSocket-only generators never reach, so its quantiles were always null. There is no rpc_methods group either (FU-4).
  • Overhead benchmarkbenchmark.sh runs the workload with telemetry off and on.
  • CI.github/workflows/telemetry-validation.yml: build → cluster → load → validate → regression gate → report.

Follow-ups

  • FU-2: publish the baseline as a develop artifact instead of committing it
  • FU-4: replace benchmark.sh proxy measurements with PromQL, and gate RPC method timings
  • FU-6: historical trend dashboard

API Impact

None — test, CI and documentation assets only.

Test Plan

  • docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld — cluster reaches consensus, workload completes.
  • Validation reports zero missing spans, attributes and metrics.
  • Regression gate passes against the committed baseline.

Comment thread .github/workflows/reusable-telemetry-validation.yml
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase10-workload-validation branch from dc53faa to d3d2325 Compare March 10, 2026 20:08
Comment thread .github/workflows/reusable-telemetry-validation.yml
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase10-workload-validation branch 3 times, most recently from 7fb4b09 to 0a44290 Compare March 11, 2026 14:54
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase10-workload-validation branch 3 times, most recently from 3faf309 to ea60b46 Compare March 11, 2026 16:58
Comment thread .github/workflows/reusable-telemetry-validation.yml
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase10-workload-validation branch from 252aceb to 42a46a3 Compare March 12, 2026 10:54
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase10-workload-validation branch from 42a46a3 to 0314b25 Compare March 12, 2026 12:00
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase10-workload-validation branch from 0314b25 to fb71866 Compare March 12, 2026 13:04

@pratikmankawde pratikmankawde left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI Build Performance Issues

The Telemetry Validation workflow is timing out because:

  1. Conan packages are being built from source (~45 min) instead of pulled as pre-built binaries from the remote
  2. No ccache for the xrpld compilation step (~45+ min cold build)

Combined, these exceed the 90-minute timeout. See inline comments for specific fixes.

@pratikmankawde pratikmankawde left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Build Performance Fixes

4 suggestions to fix the timeout. Apply all of them together:

  1. Add ccache env vars
  2. Install ccache package
  3. Authenticate to Conan remote (this is why deps build from source for 45 min)
  4. Use ccache in CMake configure

Comment thread .github/workflows/telemetry-validation.yml
Comment thread .github/workflows/telemetry-validation.yml Outdated
Comment thread .github/workflows/telemetry-validation.yml Outdated
Comment thread .github/workflows/telemetry-validation.yml

@pratikmankawde pratikmankawde left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix broken YAML from previous suggestions

The 4 suggestions below fix the duplicate/missing lines. Apply all as a batch.

@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase10-workload-validation branch from 5dfef07 to 6367ca1 Compare March 12, 2026 22:14
@codecov

codecov Bot commented Mar 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@xrplf-ai-reviewer xrplf-ai-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Four bugs in telemetry harness — see inline.

prod = References(set(), set())
tests = References(set(), set())
for path in iter_sources(root, REFERENCE_EXTENSIONS):
text = strip_comments(read_source(path))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

String literals not masked in referenced_constants() cause false references in logs. Mask like constant_definitions():

        text = strip_comments(read_source(path))
        text = STRING_LITERAL.sub(lambda m: " " * len(m.group(0)), text)
        if path.name.endswith("SpanNames.h"):
Suggested change
text = strip_comments(read_source(path))
text = STRING_LITERAL.sub(lambda m: " " * len(m.group(0)), strip_comments(read_source(path)))

XRPLD="$2"
shift 2
;;
--nodes)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NUM_NODES not validated — 0 passes silently, non-numeric crashes script with seq error:

case "$2" in
    ''|*[!0-9]*|0) die "--nodes requires a positive integer, got: ${2:-empty}" ;;
esac
NUM_NODES="$2"

try:
custom = json.loads(args.weights)
weights = {k: int(v) for k, v in custom.items()}
if not weights or sum(weights.values()) <= 0:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Negative weights allowed — random.choices() raises unhandled ValueError. Reject negative weights:

if any(w < 0 for w in weights.values()) or sum(weights.values()) <= 0:
    sys.exit(f"--weights must have non-negative values and positive sum, got: {weights}")
Suggested change
if not weights or sum(weights.values()) <= 0:
if not weights or sum(weights.values()) <= 0 or any(v < 0 for v in weights.values()):

if (pids != "" && cpu_n != n) exit 1
if (cpu_n > 0) printf("%.2f\n", cpu_sum / cpu_n) >> cpu_file
}
' || die "process sampling failed on sample $sample/$SAMPLES: expected $(echo "$PIDS_CSV" | tr -cd , | wc -c)+1 live pids"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arithmetic syntax $(wc -c)+1 appends literal '+1' not result. Wrap in arithmetic expansion:

die "expected $(($(echo \"$pid_list\" | wc -c)+1)) live pids"
Suggested change
' || die "process sampling failed on sample $sample/$SAMPLES: expected $(echo "$PIDS_CSV" | tr -cd , | wc -c)+1 live pids"
' || die "process sampling failed on sample $sample/$SAMPLES: expected $(( $(echo "$PIDS_CSV" | tr -cd , | wc -c) + 1 )) live pids"

pratikmankawde and others added 7 commits September 22, 2026 15:42
…e validator

The sanity bound required the value to be strictly positive, which rejected
three of the four readings the gauge can legitimately produce: a negative
count once the validator list has expired, the -1 sentinel for "no published
list fetched", and +inf for a config-listed list that never expires.

The negative case is the one that matters. It is the signal that expiry has
already passed, so the gauge deliberately does not clamp at zero, and a gate
that rejects it would fail exactly when an operator most needs the reading.
The -1 sentinel already violated the bound and had simply never been hit,
because the validation cluster always fetches a published list.

The floor is now a century, which still catches a broken clock. Detecting the
unsigned wrap this bound used to hide moves to the MetricsRegistry::daysUntil
unit tests, which are deterministic and do not need a running cluster.
…om Ok

This board's own copy of the Success series selected status_code=STATUS_CODE_OK.
Successful rpc.command spans no longer carry Ok: the specification reserves it
for an operator asserting verified success and warns a tool may read it as
suppressing errors, so instrumentation leaves a successful span Unset.

The blob differs from phase-9's, so merging forward would not have corrected it.
… an object

Both workload scripts decoded JSON and went straight to .get() or .items().
An array or a scalar decodes fine and then raises AttributeError, which is not a
ValueError, so the handler around the weights parsing could not catch it and the
operator saw a traceback naming neither the command nor what arrived.

Four sites across the two files, all four guarded, and the weights handler now
also catches TypeError for a non-numeric weight value.

Negative weights are left alone: they do not raise, the type is dropped from the
mix silently, and changing that changes accepted input.
The harness checked parentage only within a node. Every declared
relationship named two spans from one process, so a receive span that
lost its cross-node parent still satisfied the contract: it is created
either way, with all its normal attributes, and only its parent differs.

Declare the three pairs that cross a node boundary, so a send side that
stops propagating context fails the run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@xrplf-ai-reviewer xrplf-ai-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unhandled exception bypasses stats recording—see inline.

# AttributeError that follows names neither the command nor what
# arrived. Reject it here so the message carries both.
if not isinstance(reply, dict):
raise TypeError(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TypeError escapes exception handler, corrupts stats—use AttributeError:

Suggested change
raise TypeError(
raise AttributeError("expected dict")
Suggested change
raise TypeError(
raise AttributeError(

pratikmankawde and others added 17 commits September 23, 2026 12:00
The $xrpl_work_item variable's description carried a real ticket id as its
example. The filter needs no example, so the id is gone and the wording stays.
The $xrpl_work_item variable's description carried a real ticket id as its
example. The filter needs no example, so the id is gone and the wording stays.
clang-tidy's readability-math-missing-parentheses flags seven mixed
precedence expressions in this file, and warnings are errors in CI.

The parentheses match the precedence the compiler already applied, so every
static_assert value and the loop bound are unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only SpanGuard carried addEvent(name, attrs), so a call site holding a scoped
guard could not record an event attribute. Forwarding overload, with the no-op
twin in the telemetry-disabled stub, so a span can be converted between scoped
and unscoped without dropping the attributes on its events.
accept.apply was a plain SpanGuard, so it never became the ambient span of
doAccept. The spans the function goes on to create inherited the activated
accept span instead and came out as accept.apply's siblings, while running
inside its own time window. Every guard was scoped before the SpanGuard split,
so this restores the hierarchy that design had.

Scoped now, so the hierarchy follows the call flow. Drops the parent-context
fallback arm with it: the accept context is captured only while the accept span
is live, and that span is a child of the round context, so an invalid accept
context implies an invalid round context and both arms returned an empty guard.
MonitoredMode::set calls onModeChange on every round start, so the span was
created whether or not the mode moved. A node with a steady mode therefore
emitted one mode_change per round carrying mode_old == mode_new, which a live
sweep confirmed on every round of both instrumented builds. The round span's
own consensus_mode attribute is still written on every call, since that is
where the round learns the mode it is running in.
… harness

The contract's parent field was documentation only, so no check could fail a
span for being parented when it should be a root. A live sweep found three
spans nested one level off while every harness check passed. The field becomes
allowed_parents, a list derived per span from its creation factory and every
call path that reaches it, and is asserted per span.

A parent on another node and a parent absent from the trace are both
inconclusive rather than violations: the receive spans are parented cross-node
by design, and a rotation in flight has not exported its root yet. Spans
reached by two call paths list every lawful parent.

A second gate reads a round as a unit: every required phase child under the
round span on the same node, their start times in protocol order, and no
mode_change recording mode_old == mode_new. It selects traces that already hold
the accept span, since that span always ends after the round span and can land
in a later export batch.

num_cleared leaves txq.batch_clear's required attributes: the code writes it
only after the batch actually clears.
… arm went

The accept.apply factory no longer falls back to the round context, so the
harness contract lists consensus.accept alone. Note rewritten to describe the
scoped child as it is now.

@xrplf-ai-reviewer xrplf-ai-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dead constant detection incomplete, exception handling missing in load generator—see inline.

name when it sits outside any namespace and so has no qualified form."""
if qualified in refs.pairs:
return True
return qualified == name and name in refs.bare

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Namespace-less dead constants aren't detected: refs.bare includes the definition. Exclude self-references:

# json.JSONDecodeError - reply body was not valid JSON.
# AttributeError - reply parsed to something with no .get(), e.g. a
# JSON array.
REQUEST_FAILURES: tuple[type[BaseException], ...] = (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TypeError missing from REQUEST_FAILURES when non-dict replies occur—stats not recorded. Add it to the tuple:

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DistributedTracingAndObservability Distributed Tracing And Observability related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants